Please type a plus sign (+) inside this box [+] pto/sb/05 {12/97) 

Approved for use through 09/30/00. OMB 0651-0032 
Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCE 

Under the Paperwoil^ Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number 



UTILITY PATENT APPLICATION TRANSMITTAL 

(Only for new nonprovisional applications under 37 CFR 1 .53(b) 

Attorney Docket No. 80398.P350 Total Pages _2_ 

First Named Inventor or Application Identifie r Ted J. Cooper 

Express Mail Label No. EL672752346US 



in en 



ADDRESS TO: Assistant Commissioner for Patents 
Box Patent Application 
Washington, D. C. 20231 



APPLICATION ELEMENTS 

See MPEP chapter 600 concerning utility patent application contents. 

1 . X Fee Transmittal Fornn 

(Submit an original, and a duplicate for fee processing) 

2. X Specification (Total Pages 23 ) 

(preferred arrangement set forth below) 

- Descriptive Title of the Invention 

- Cross References to Related Applications 

- statement Regarding Fed sponsored R&D 

- Reference to Microfiche Appendix 

- Background of the Invention 

- Brief Summary of the Invention 

- Brief Description of the Drawings (if filed) 

- Detailed Description 

- Claims 

- Abstract of the Disclosure 

3. X Drawings(s) (35 USC 1 1 3) (Total Slieets 10 ) 

4. X Oath or Declaration (Total Pages _4 ) 

a. X Newly Executed (Original or Copy) 

b. Copy from a Prior Application (37 CFR 1 .63(d)) 

(for Continuation/Divisional with Box 17 completed) (Note Box 5 below) 

i. DELETIONS OF INVENTOR(S) Signed statement attached deleting 

inventor(s) named in the prior application, see 37 CFR 1 .63(d)(2) 
and 1.33(b). 

5. Incorporation By Reference (useable if Box 4b is checked) 

The entire disclosure of the prior application, from which a copy of the oath or 
declaration is supplied under Box 4b, is considered as being part of the 
disclosure of the accompanying application and is hereby incorporated by 
reference therein. 

6. Microfiche Computer Program (Appendix) 



12/01/97 



PTO/SB/05 (12/97) 



7. Nucleotide and/or Amino Acid Sequence Submission 

(if applicable, all necessary) 

a. Computer Readable Copy 

b. Paper Copy (identical to computer copy) 

c. Statement verifying identity olf above copies 

ACCOMPANYING APPLICATION PARTS 

8. X Assignmeint Papers (cover sheet & documents(s)) 

9. a. 37 CFR 3.73(b) Statement (where there is an assignee) 

X b. Power of Attorney 

1 0. English Translation Document (if applicable) 

1 1 . a. Information Disclosure Statement (IDS)/PT0-1 449 

b. Copies of IDS Citations 

12. Preliminary Amendment 

1 3. X Return Receipt Postcard (MPEP 503) (Should be specifically itemized) 

14. a. Small Entity Statement(s) 

b. Statement filed in prior application, Status still proper and desired 

1 5. Certified Copy of Priority Document(s) (if foreign priority is claimed) 

16. _X Other: Certificate of Express Mail with copy of postcard showino contents of 

Express Mail packaoe. 



17. If a CONTINUING APPLICATION, check appropriate box and supply the requisite information: 

Continuation Divisional Continuation-in-part (CIP) 

of prior application No: 



1 8. Correspondence Address 

Customer Number or Bar Code Label 



or 

J( Correspondence Address Below 



(Insert Customer No. or Attach Bar Code Label here) 



NAME Sang Hui Michael Kim 



BLAKELY. SQKOLOFF. TAYL(DE{ & ZAFMAN LLP 



ADDRESS 12400 Wilshire Boulevard 



Seventh Floor 



CITY LosAnoeles STATE California ZIP CODE 90025-1026 



Country U.S.A. TELEPHONE (408^ 720-8300 FAX (408^ 720-9397 



12/01/97 



PTO/SB/05 (12/97) 
Approved for use through 09/30/00. OMB 0651-0032 
Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE 



PTO/SB/1 7(08-00) 

Approved for use through 10/31/2002. 0MB 0651-0032 
Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCE 

Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid 0MB control number. 

FEE TRANSMITTAL FOR FY 2001 

TOTAL AMOUNT OF PAYMENT ($) ■ 830,00 

Complete if Known: 

Application No. Not vet assigned 

Filing Date Herewith 

First Named Inventor Ted J. Cooper 

Group Art Unit Not vet assigned 

Examiner Name Not vet assigned 

Attorney Docket No. 80398.P350 

METHOD OF PAYMENT (check one) 

1 ■ [ X ] The Commissioner is hereby authorized to charge indicated fees and credit 
any over payments to: 

Deposit Account Number 02-2666 

Deposit Account Name 

[ X ] Charge Any Additional Fee Required Under 37 CFR 1.16 and 1 .17 



2. X Payment Enclosed: 

_X Check 

Money Order 

Other 



FEE CALCULATION 

1- BASIC FILING FEE 



Larae 


Entity 


Small 


Entity 




Fee 


Fee 


Fee 


Fee 




Code 


($) 


Code 


($) 


Fee DescriDtion 


101 


710 


201 


355 


Utility application filing fee 


106 


320 


206 


160 


Design application filing fee 


107 


490 


207 


245 


Plant filing fee 


108 


710 


208 


355 


Reissue filing fee 


114 


150 


214 


75 


Provisional application filing fee 



Fee Paid 

710.00 



SUBTOTAL (1) $ 710-00 



2. EXTRA CLAIM FEES 



Total Claims 20 



Extra Claims 



-20** = 
-3** = 



Independent Claims 4 
Multiple Dependent 
Or number previously paid, if greater; For Reissues, see below. 



X 
X 



Fee from 

below Fee Paid 

18.00 = 

80.00 = 80.00 



Larqe 


Entity 


Small 


Entity 


Fee 


Fee 


Fee 


Fee 


Code 


{$) 


Code 


($) 
9 


103 


18 


203 


102 


80 


202 


40 


104 


270 


204 


135 


109 


80 


209 


40 


110 


18 


210 


9 



Fee Description 

Claims in excess of 20 

Independent claims In excess of 3 

Multiple dependent claim, if not paid 

**Reissue independent claims over original patent 

**Relssue claims in excess of 20 and over original patent 

SUBTOTAL (2) $ 80.00 



10/01/2000 



FEE CALCULATION (continued) 




3. 


ADDITIONAL FEES 






Larae Entity 


Small Entity 






Fg© 


Fee 


Fee 


ree 






uode 


(S) 


Code 


(5) 


Fee DescrlDtion 


Fee Paid 


'I nc 
1 UO 


loU 


2UO 


DO 


Surcharge - late filing fee or oath 








£.£.( 




Surcharge - late provisional filing fee 












or cover sheet 






loO 


A OO 


130 


Non-English specification 




147 


2,520 




2,520 


For filing a request for reexamination 




112 


920* 


112 


920* 


Requesting publication of SIR prior to 












Examiner action 






1,o40 


113 


1 ,840 


Requesting publication of SIR after 












Examiner action 






A At\ 
1 10 


215 


55 


Extension for response within first month 




no 


390 


216 


195 


Extension for response within second month 




'4-17 
11/ 


o90 


217 


445 


Extension for response within third month 




118 


1,390 


218 


695 


Extension for response within fourth month 




A OO 

128 


1,890 


228 


945 


Extension for response within fifth month 




119 


310 


219 


155 


Notice of Appeal 




120 


310 


220 


155 


Filing a brief in support of an appeal 




121 


270 


221 


135 


Request for oral hearing 




138 


1,510 


138 


1,510 


Petition to institute a public use proceeding 




140 


110 


240 


55 


Petition to revive unavoidably abandoned 












application 




141 


1,240 


241 


620 


Petition to revive unintentionally 












abandoned application 




142 


1,240 


242 


620 


Utility issue fee (or reissue) 




143 


440 


243 


220 


Design issue fee 




144 


600 


244 


300 


Plant Issue fee 




122 


130 


122 


130 


Petitions to the Commissioner 




123 


50 


123 


50 


Petitions related to provisional applications 




126 


240 


126 


240 


Submission of Information Disclosure Stmt 




581 


40 


581 


40 


Recording each patent assignment per 












property (times number of properties) 


40.00 


146 


710 






For filing a submission after final rejection 












^see of K^rn imi£.^\Ja)) 




149 


710 


249 


355 


For each additional Invention to be examined 












(see 37 CFR 1.129(b)) 




179 


710 


279 


355 


Request for Continued Examination (RCE) 




169 


900 


169 


900 


Request for expedited examination of a design 












application 




Other fee (specify) 










Other fee (specify) 


















SUBTOTAL (3) 


$ 40 on 


^Reduced by Basic Filing Fee Paid 




SUBIVUTTED BY: 










Typed or Printed Name: 


Sana Hui Michael Kim 




Slgnature:,^^^ 






^ Date: /g/^Y/oO 




Reg. Number: 


4^50 




Telephone Number: 

















10/01/2000 



-2- 



PTO/SB/17(10/00) 



Attorney Docket No.: 80398.P350 



PATENT 



UNITED STATES PATENT APPLICATION 
For 



METHOD AND APPARATUS TO PROVIDE EDGE ENHANCEMENTS AS 
PART OF A DEMOSAICING PROCESS 



Inventor: 
Ted J. Cooper 



Prepared By: 

Blakely, Sokoloff, Taylor & Zafman llp 
12400 WiLSHiRE Boulevard 

Seventh Floor 
Los Angeles, C A 90025-1026 

(408) 720-8300 



"Express Mail" mailing label number 
Date of Deposit: _ 

I hereby certify that I am causing this paper or fee to be deposited with the 
United States Postal Service "Express Mail Post Office to Addressee" service 
under 37 CFR 1.10 on the date indicated above and is addressed to the 
Commissioner of Patents and Trademarks, Washington, D.C. 20231 



;ioner or I'atents ana iraaemarKs, wasnmgion, j 



(Typed or printed name of pers(5fiinailing^per or fee) 




{Signature of person mailing paper or fke) Date 



METHOD AND APPARATUS TO PROVIDE EDGE ENHANCEMENTS AS 
PART OF A DEMOSAICING PROCESS 

FIELD OF THE INVENTION 

The present invention relates generally to image processing and image processing 
5 devices. More particularly, the present invention relates to a method and apparatus to 
provide edge enhancements as part of a demosaicing process. 
BACKGROUND OF THE INVENTION 

A common image processing device is a digital still camera (DSC). A DSC is an 
image input device that typically includes a single charge-couple device (CCD) array. A 
10 CCD array is an array of light detectors, which typically detects one of three monochrome 
colors (e.g., red R, green G, or blue B) for each light detector location. To output a color 
image, however, the DSC must provide a complete set of RGB (or CMY or YCrCr) tri- 
stimulus values for each light detector location. Such a process is commonly referred to 
as "demosaicing" in which full color at every pixel is calculated from a patchwork of 
1 5 color filter values distributed over the captured image. 

Thus, a typical image processing workflow for a current DSC is to capture an 
image using the CCD array of light detectors, to convert the information from the light 
detectors into digital form (i.e., raw captured image data), and to provide color to image 
data by a demosaicing process, to perform a white balancing process, to perform a 
20 chromatic improvement process, and finally to perform an edge enhancement process. 

ha the image capturing process, prior to demosaicing, a DSC arranges the colors 
from the light detectors in a systematically repeating pattern as the raw captured image 
data. For example, the raw captured image data can have "Bayer" pattern with 
interleaved lines of "RGRGRG. . . " followed by lines of "GBGBGB ..." At this point, the 
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captured image is represented as a mosaic of RGB color primaries for each light detector 
location in which no color can be viewed. 

Demosaicing is thus the process of computing a complete set of tri-stimulus 
values at every CCD light detector location to provide color to the image. A number of 
5 algorithms can be used to compute the complete set of values, which typically involve 
nearest neighbor substitution, bi-linear interpolation, and median filtering. For example, 
a pixel or light detector location, which corresponds to a green detector can infer from 
neighboring detectors what red and blue values should be at that location where the green 
detector is located. Similarly, a pixel that corresponds to a red detector can infer from 

10 neighboring detectors what the green and blue values are at that location. 

A limitation with the demosaicing process, which is equivalent to low-pass 
filtering process, is that it has some inherent side effects. For example, a low-pass 
filtering process attenuates the high-frequency detail or accentuates the low-frequency 
detail of an image. Thus, a common side effect is that it introduces chromatic fringing at 

15 the edges of sharp contrast boundaries in an image. That is, the edges of the sharp 
contrast boundaries have red and blue fringe artifacts. FIG. 1 shows an image 100 
having such red and blue artifacts referenced by numerals 101 and 103, respectively. 

Currently, the DSC industry uses various forms of edge enhancement processes to 
improve an image. An edge enhancement process is used to detect edges in an image and 

20 to increase the edge detail of the image to produce a more realistic and sharper image in 
keeping with certain aspects of the Human Visual System (HVS). Such a process, 
however, compounds the problem of the red and blue fi'inge artifacts because the edge 
enhancement typically causes the magnitude of the color fi-inging to increase in direct 
proportion to the intensity of the enhancement. 
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Furthermore, a disadvantage with the operation of a current DSC is that edge 
enhancements occur at the end of the image processing workflow, i.e., after white 
balancing and chromatic enhancement processes. White balancing is a process used to 
calibrate the proportion of red, green, and blue color values so that the white color in the 
5 image results in a pure white color without color casts. Chromatic enhancement is a 
process used to provide a deeper or more saturated color to the color image. Both of 
these processes, however, alter the raw captured image data before the edge 
enhancements can take place. 

Consequently, a great deal of spatial and chromatic information originally present 
10 in the raw image data may have been before the edge enhancement process could have 
access to the data. This may cause low color fidelity and degraded edge sharpness. For 
example, as illustrated in FIG. 1, an exemplary image 100 is shown in which edge 
enhancements are performed late in an image processing workflow in accordance with 
prior art. 

15 Referring to FIG. 1, image 100 shows a number "2," which is greatly magnified 

for purposes of illustration. Blue and red pixel alternations can be noticed by reference to 
numberals 101 and 103 in image 100 along the number "2." Such alternations are 
commonly referred to as a "chromatic moire." The chromatic moire is greatly 
exaggerated because edge enhancements occur late in the image data processing 

20 workflow thus causing the image to have low color fidelity and degraded edge sharpness. 
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SUMMARY OF THE INVENTION 

According to one aspect of the invention, an image processing method is 
disclosed in which an image is captured and edge enhancements are provided to the 
captured image as part of a demosaicing process. 

Other features and advantages of the present invention will be apparent from the 
accompanying drawings, and from the detailed description, which follows below. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The features and advantages of the present invention are illustrated by way of 
example and not intended to be limited by the figures of the accompanying drawings, in 
which like references indicate similar elements and in which: 

FIG. 1 depicts an exemplary image in which edge enhancements are performed 
late in an image processing workflow in accordance with prior art. 

FIG. 2 depicts an exemplary image processing system in which the present 
invention can be implemented; 

FIG. 3 depicts a flow chart illustrating an image processing operation according 
to one embodiment; 

FIG. 4 depicts a flow chart illustrating post demosaicing processing according to 
one embodiment; 

FIG. 5 depicts a flow chart illustrating a detailed operation for providing edge 
enhancements as part of the demosaicing process of FIG. 3 according to one 
embodiment; 

FIG. 6A depicts an exemplary schematic image based on raw data; 
FIG. 6B depicts an exemplary schematic gray map image; 
FIG. 6C depicts an exemplary high contrast masked edge for a mask image; 
FIG. 6D depicts an exemplary image with unsharp edge enhancement; 
FIG. 6E depicts an exemplary luminance graph of a cross-sectional view of the 
image of FIG. 6D; 

FIG. 6F depicts an exemplary schematic image with multiplicative blending; 
FIG. 6G depicts an exemplary image in which edge enhancments are provided as 
part of a demosaicing process; 
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FIG. 7 depicts a flow chart illustrating an operation for a creating mask image 
from a fringe corrected image according to one embodiment; 

FIG. 8A depicts an exemplary a 5 x 5 pixel pattern with the central pixel being a 
part of the edge of an image; and 

FIG, 8B depicts an exemplary L*<3*6* color space chart. 
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DETAILED DESCRIPTION 

According to the embodiments described herein, a method and apparatus are 
described in which an image is captured and edge enhancements are provided to the 
captured image as part of a demosaicing process. By providing edge enhancements as 
5 part of the demosaicing process, significant improvements in color fidehty and edge 
sharpness are possible because the edge enhancement process has access to unaltered 
spatial and chromatic information contained in the raw captured image data. 

FIG. 2 depicts an exemplary image processing system 200 in which the present 
invention can be implemented. The image processing system 200 can represent any 
10 number of image processing devices such as, for example, an electronic camera, video 
camera, or a digital still camera (DSC), and is not intended to be limited to any particular 
image processing device. 

Referring to FIG. 2, image processing system 200 includes a charge-couple 
device (CCD) array 210, an analog/digital (A/D) converter 220, memory storage 230, 
1 5 processor 240, and output 250. The CCD array 210 is coupled to the A/D converter 220. 
The A/D converter 220 is coupled to the memory storage 230. The processor 240 and 
output 250 are coupled to the memory storage 230. 

CCD array 210 is an image detector having an array of Ught detectors. The CCD 
array 210 is only one type of image detector that may be used, however, other alternative 
20 image detectors can be used. For example, phototransistors, photodiodes, or any other 
form of photo capture device can be use as an image detector. CCD array 210 acquires 
raw image data using the array of light detectors. For one embodiment, CCD array 210 
detects red R, green G, and blue B color primaries and arranges the color primaries in a 
repeating pattern such as, for example, a "Bayer" pattern, as "raw captured image data." 
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For an alternative embodiment, CCD array 210 detects cyan-magenta-yellow-green 
(CMYG) color primaries and arranges the color primaries in a repeating pattern as the 
raw captured image data. 

A/D converter 220 is a processing device that converts analog signals into digital 
5 signals. A/D converter 220 receives image information from the CCD array 210 as 

analog signals and converts them into digital signals. The digital signals representing the 
captured image are stored in memory storage 230 as the raw image data. 

Memory storage 230 is a memory device to store data or information. Memory 
storage 230 can be a random access memory (RAM), flash memory, or other like memory 
10 devices. For alternative embodiments, memory storage 230 can include a plurality of 
memory devices. Memory storage 230 stores image data or information from the CCD 
array 210 and A/D converter 220. Alternatively, memory storage 230 can also store 
instructions to be processed by processor 240 to perform image processing operations on 
image data. Furthermore, memory storage 230 can provide short-term and long-term 
1 5 storage of image data for processing, display, and archival purposes. 

Processor 240 is a processing device such as a high-speed microprocessor. 
Processor 240 can access memory storage 230 and process image data stored therein. For 
one embodiment, processor 240 processes raw image data stored in memory storage 230 
and provides edge enhancements to the raw image data as part of a demosaicing process 
20 as will be described in further detail below. Processor 240 can also store the edge 

enhanced image back in memory storage 230. Furthermore, processor 240 can cause the 
edge enhanced image to be sent to output 250 or to be further processed. 

Output 250 can be a display device such as, for example, a liquid crystal display 
(LCD) or other like display device. Alternatively, output 250 can be another storage 

8 80398.P350 



device or a data compression device. Furthermore, output 250 can be an interface to an 
external or remote device in which image data is displayed or provided to the external or 
remote device. Output 250 can also output information or data to the external or remote 
device by a direct connection or a network connection. 
5 The image processing techniques described herein can be implemented by 

hardware and/or software contained within image processing system 200. For example, 
image processor 240 can execute code or instructions stored in a machine-readable 
medium, e.g., memory storage 230, to perform the operations as described in FIGS. 3, 4, 
5, and 7. 

10 The machine-readable medium may include a mechanism that provides (i.e., 

stores and/or transmits) information in a form readable by a machine such a processor, 
computer, or a digital processing device. For example, a machine-readable medium may 
include a read only memory (ROM), random access memory (RAM), magnetic disk 
storage media, optical storage media, or flash memory devices. The code or instructions 

15 can be represented by carrier wave signals, infrared signals, digital signals, and by other 
like signals. 

FIG. 3 depicts a flow chart illustrating an image processing operation 300 
according to one embodiment. The operation 300 illustrates combining "demosaicing" 
and edge enhancement in the same process. 
20 For purposes of explanation, operation 300 begins at operation 310. Referring to 

FIG. 3, at operation 310, an image is captured. For example, the CCD array 210 can 
capture an image using an array of light detectors. The captured image can be stored in 
memory storage 230 as raw image data in a Bayer pattem. Altematively, the raw image 
data can be captured in other patterns. 
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At operation 350, edge enhancements are provided to the captured image as part 
of a demosaicing process. For example, processor 240 can process the raw image data 
stored in memory storage 230 to provide edge enhancements as part of a demosaicing 
process such as that explained with regards to FIG. 5, Alternatively, any form of edge 
5 enhacnement process can be used for operation 300 such that it is performed as part of 
demosaicing. 

By providing edge enhancements early in an image processing workflow, (i.e., as 
part of the demosaicing process), there can be significant improvements in color fidelity 
and edge sharpness because the edge enhancement process has access to unaltered spatial 
10 and chromatic information contained in the raw captured image data. 

FIG. 4 depicts a flow chart illustrating post demosaicing processing according to 
one embodiment. The following operations shown in FIG. 4 can be performed after 
operation 350 of FIG. 3. Alternatively, the following operations can be optional. 

Referring to FIG. 4, at operation 360, a post demosaicing process can be 
1 5 performed. For example, processes to further improve the image and color can be 
performed such as white balancing or chromatic improvement. Alternatively, the 
demosaiced image can be compressed to be stored in memory storage 230. 

At operation 370, the processed color image can be output to a user via a visual 
display. Alternatively, the color image can be output to a storage device for storage. The 
20 color image can also be output to external and remote devices either by a direct 
connection or a network connection. 

FIG. 5 depicts a flow chart illustrating the detailed operation 350 for providing 
edge enhancements as part of the demosaicing process of FIG. 3 according to one 
embodiment. The operation 350 not only provides edge enhancement, but also 
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demosaicing. Thus, operation 350 illustrates an operation that provides edge 
enhancements as part of a demosaicing process. For purposes of explanation, operation 
350 begins at operation 510. 

Referring to FIG. 5, at operation 5 10, a gray map (or brightness map) using the 
5 raw captured image data (e.g., RGB data) is created. FIG. 6A depicts an exemplary 

schematic image based on raw data. A demosaicing process may be employed to extract 
a first order approximation for the R, and B pixel values of the raw data to create a 
gray map (e.g., bi-linear interpolation). The gray map is an image composed of gray-level 
brightness. FIG. 6B depicts an exemplary schematic gray map image. It should be noted 
10 that attempts using a simplified luminance approximation for the gray map found in 
Equation (1) below failed noticeably if chromatic edges are present. 

Luminance -ft 7 77*i? +0 [failed] (1) 
Instead, a luminance based on L*a*Z?* is much more robust. However, it is not 

required to use the fiill CIE L* calculation. The L*a*6* is one way to represent colors in 
15 color space, in which L* represents brightness, and a* and 6* define chromatic axes. The 

luminance based on L*a*Z?* is used because it provides a perceptually uniform color 

space. The RGB data can easily be converted to L*a*^?* values for some assume image 

white point. 

At operation 520, the gray map image is stored. For example, the gray map image 
20 such as that shown in FIG. 6B can be stored in storage memory 230. 

At operation 530, a mask image is created, which is formed by detecting edges in 
the gray map and providing enhancements to the edged. The edge detection/enhancement 
process is discussed in further detail in FIG. 7. For one embodiment, the pixels of the 
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mask image are different from the original image located in strong brightness gradient 
areas. The final "color" of each changed pixel is either a maximum or minimum form the 
pixels that surround it. It should be noted that any form of edge enhancement can be used 
in this invention. Current methods include unsharp masking, zero-crossing gradient 
5 detection, chromatic discontinuity, and various kernel transforms. The use of the mask 
technique described here is just one embodiment of the edge enhancement process. 

FIG. 6C depicts an exemplary high contrast masked edge for a mask image. 
For one embodiment, FIG, 6C may illustrate the mask image generated from FIG. 6B. 
Although no color is shown in FIG. 6C, the mask image can have fewer colors and more 

10 smoothly varying colors than the original image. Moreover, the visibiUty of the textual 
edges is more pronounced and lacks the chromatic moire around the slanted edges of the 
text. This can have the affect of providing a block-like appearance. 

At operation 540, unsharp edge enhancement is performed on the mask image. 
That is, the block-like structured edges are smoothed by an unsharp edge enhancement 

1 5 process, which changes the luminance level to accentuate the edge transition contrast of 
an image. Also, because the mask image has strong visual steps in its contour, it typically 
needs to be smoothed by adding a fractional part of the original demosaiced image to it. 
FIG. 6D depicts an exemplary image with an unsharp edge enhancement, and FIG. 6E 
depicts an exemplary luminance graph of a cross-sectional view of the image of FIG. 6D. 

20 At operation 550, multiplicative blending is performed to obtain the final output. 

For example, a blending process based on the formula (2) below can be performed to 
obtain the desired output. 

Output = Mask ^ (1-a) + Original * a (2) 
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This blending process yields the original demosaiced RGB values everywhere 
except where strong gradients were found. FIG. 6F shows an exemplary image with the 
multiplicative bending. The optimal value of a depends on use for the image. For 
example, if soft smooth edges necessary, then larger values of a are preferred. 
5 Furthermore, for images with small text or fine jewelry, smaller values of a are preferred. 

For one embodiment, FIG. 6D and FIG. 6A or 6B may be blended 
multiplicatively to obtain the final image. FIG. 6G depicts an exemplary final output 
image in which edge enhancement was provided during a demosaicing process. By way 
of contrast to FIG. 1 in which edge enhancement is performed at the end of an image 
10 processing workflow, FIG, 6G shows significant improvement in color fidelity and edge 
sharpness. This is a result of performing edge enhancement as part of the demosaicing 
process. 

FIG. 7 depicts a flow chart illustrating the detailed operation 530 of FIG. 5 to 
create a mask image by performing edge detection/enhancement. The operation 530 uses 
1 5 the knowledge of the edge location to sharpen the edges of an image in a non-linear way 
to produce contrast enhancement. For purposes of explanation, operation 530 begins at 
operation 710. 

At operation 710, an NxN pixel neighborhood of a central pixel, which is part of a 
detected edge, is investigated. To determine the central pixel, a threshold criteria is 
20 applied. For example, a threshold criteria is applied to each pixel to determine if it is 

involved in a significant brightness change compared to its neighbors. Exemplary values 
for the threshold range from 20-35 units based on an 8-bit monochrome image scale (after 
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the gray image has been created). Thus, once the central pixel is determined to have 
passed the threshold criteria, the operation investigates the NxN pixel neighborhood. 

The central pixel (each of the pixels that are determined as a part of the edges) is 
examined in terms of a localized matrix or cell of pixels that surround the pixel in 
5 question in a square array whose size is N pixels v^ide by N pixels high. Exemplary 
values of N can be odd numbers, such as 3, 5, 7.. . to N, However, a 5 x 5 pixel is 
preferred because it calculates in all the standard compass directions without 
overwhelming the image processing device's performance. FIG. 8A illustrates 
schematically a 5 x 5 pixel with the central pixel being a part of the edge of an image. 
10 At operation 720, each of the twenty-four pixels in the neighborhood of the 

central pixel is ranked based on its brightness and perceptual color values. To rank each 
pixel, L*a*6* values are used. The L*a*^* values can be converted from RGB values 
mathematically. 

FIG. 8B shows an L*a*Z?* color space chart divided into multiple bins to classify 
15 the color values for each pixel. For one embodiment, the chart is divided into eight bin 
spaces, but the chart can be divided more than eight bins for greater accuracy. The a* 
axis runs from green (+a*) to magenta (-a*), and the Z?* axis runs from yellow (+6 *) to 
blue (-^*). 

The chart is further divided into two regions - a first region, which is anywhere in 
20 between +4 units and -A units (based on 8-bit storage values for a* and b*) on either the 
a* or &* axis (shaded region in FIG. 8 A), and a second region that is anywhere outside of 
the first region. The color values of the first region are neutral and any pixel that falls 
within the first region are grouped and handled separately than any pixel that falls within 
the second region. The pixels whose chromaticity (color amount) fall within the radius of 
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+ or - 4 units in the a* and b* axis are basically neutral colors and are excluded from 
further processing in determining the chromatic content of the neighborhood around the 
central pixel. 

At operation 730, as for the pixels that fall within in the second region, each of the 
5 twenty-four pixels is determined to be in first and second most popular color groups. 
This determination is based within the eight bin spaces according to its value. 

At operation 740, the average brightness of each of first and second color groups 
is determined. At operation 750, a test is applied to compare the central pixel brightness, 
which passed the threshold criteria, against the first and second popular color groups. 
1 0 The process checks to see if the central pixel's brightness is closer to the first most 
popular color group. 

At operation 760, if the central pixel's brightness is closer to the first most 
popular color group, then the color values for the first most popular color group are stored 
in the RGB values of central pixel in question. 
1 5 At operation 770, if the central pixel's brightness is not closer to the first most 

popular color group's average brightness (i.e., the central pixel's brightness is closer to 
the second most popular color group's average brightness), then that second most popular 
color group's color values are stored into the RGB value for the central pixel. However, 
the brightness of the central pixel is kept with its original L* value. 
20 At operation 780, the operations 710 to 770 are repeated until all the detected 

edges are processed. Thus, edge detection and enhancement are provided as part of the 
demosaicing process. 

For any neighborhood pixel that falls within the first region, it is grouped 
separately and handled as though they are their own separate groups. For any other pixel 

15 80398.P350 



in the image that is not involved in edges, the original first order demosaiced RGB values 
are used. 

The final image of the above operations in FIGS. 5 and 7 can be illustrated by 
image 600 of FIG, 6G in which edge enhancement are provided as part of a demosaicing 
5 process. By creating a mask image in the above manner, the final image can be more 
chromatically accurate and spatially correct because edge detection and enhancement are 
provided as part of the demosaicing process 

Thus, a method and apparatus to provide edge enhancement during a demosaicing 
process have been described. In the foregoing specification, the invention has been 
10 described with reference to specific exemplary embodiments thereof. It will, however, be 
evident that various modifications and changes may be made thereto without departing 
from broader spirit and scope of the invention as set forth in the appended claims. The 
specification and drawings are, accordingly, to be regarded in an illustrative sense rather 
than a restrictive sense. 
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CLAIMS 

What is claimed is: 

1 L An image processing method comprising: 

2 capturing an image; and 

3 providing edge enhancements to the captured image as part of a demosaicing 

4 process. 

1 2. The method of claim 1, further comprising: 

2 performing post demosaicing processing on the captured image; and 

3 outputting the processed image. 

1 3. The method of claim 1 , wherein providing the edge enhancements includes: 

2 creating a brightness map of the captured image. 

1 4. The method of claim 3, wherein providing the edge enhancements further 

2 includes: 

3 detecting edges of the captured image using the brightness map; 

4 creating a mask image form the edge detected brightness map; and 

5 performing unsharp edge enhancement from the masked image. 
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1 5. The method of claim 4, wherein providing the edge enhancements further 

2 includes: 

3 blending multiplicatively the unsharp edge enhanced image with the brightness 

4 map. 

1 6. An apparatus comprising: 

2 an image capturing device to capture an image; and 

3 a processor to provide edge enhancements to the captured image as part of a 

4 demosaicing process, 

1 7. The apparatus of claim 6, wherein the processor is to perform post demosaicing 

2 processing on the captured image and to output the processed image. 

1 8. The apparatus of claim 6, wherein the processor is to create a brightness map of 

2 the captured image. 

1 9. The apparatus of claim 8, wherein the processor is to detect edges of the 

2 captured image using the brightness map, to create a mask image form the edge detected 

3 brightness map, and to perform unsharp edge enhancement from the masked image. 

1 10. The apparatus of claim 9, wherein the processor is to blend multiplicatively the 

2 unsharp edge enhanced image with the brightness map. 
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1 11. A machine-readable medium that provides instructions, which if executed by a 

2 processor, cause the processor to perform the operations comprising: 

3 capturing an image; and 

4 providing edge enhancements to the captured image as part of a demosaicing 

5 process. 

1 12. The machine-readable medium of claim 1 1 , further providing instructions, 

2 which if executed by the processor, cause the processor to perform the operations 

3 comprising: 

4 performing post demosaicing processing on the captured image; and 

5 outputting the processed image. 

1 13. The machine-readable medium of claim 11, further providing instructions, 

2 which if executed by the processor, cause the processor to perform the operations 

3 comprising: 

4 creating a brightness map of the captured image. 

1 14. The machine-readable medium of claim 13, further providing instructions, 

2 which if executed by the processor, cause the processor to perform the operations 

3 comprising: 

4 detecting edges of the captured image using the brightness map; 

5 creating a mask image form the edge detected brightness map; and 

6 performing unsharp edge enhancement from the masked image. 
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1 15. The machine-readable medium of claim 14, further providing instructions, 

2 which if executed by the processor, cause the processor to perform the operations 

3 comprising: 

4 blending multiplicatively the unsharp edge enhanced image with the brightness 

5 map. 

1 16. An image processing device comprising: 

2 an image capturing unit to capture an image; 

3 a memory device to store the captured image; 

4 an output unit coupled to the memory device; and 

5 a processor to provide edge enhancements to the captured image in the memory 

6 device as part of a demosaicing process and to cause the enhanced image to be output is 

7 to the output unit. 

1 17. The image processing device of claim 16, wherein the image capturing unit 

2 includes a charge-couple device (CCD) array, phototransistors, or photodiodes. 

1 18. The image processing device of claim 16, wherein the output unit is a display 

2 device. 

1 19. The image processing device of claim 1 8, wherein the processor is to perform 

2 post demosaicing processing on the captured image and to cause the image to be output 

3 to the display device. 
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1 20. The image processing device of claim 19, wherein the post demosaicing 

2 processing is a white balancing processing or a chromatic improvement processing. 
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ABSTRACT OF THE DISCLOSURE 

A method and apparatus to provide edge enhancements as part of a demosaicing 
process are disclosed in which an image is captured, and edge enhancements are 
provided to the captured image as part of a demosaicing process. By providing edge 
enhancements as part of the demosaicing process, significant improvements in color 
fidelity and edge sharpness are possible because the edge enhancement process has 
access to unaltered spatial and chromatic information contained in the raw captured 
image data. 
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Title 37, Code of Federal Regulations, Section 1 .56 
Duty to Disclose Information Material to Patentability 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office ail information known to that individual 
to be materia! to patentability as defined in this section. The duty to disclosure information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit information which is not material 
to the patentability of any existing claim. The duty to disclosure all Information known to be material to 
patentability is deemed to be satisfied if all information known to be material to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1 .98. However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1 ) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1 ) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there Is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing infomnation to the attorney, agent, or inventor. 
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